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DETAILED ACTION 

This action is in response to the amendment filed 1 3 April 2009. Claims 1 -7 have been 
canceled. Claims 8, 11, 13, 14, 16, 17,19-21 have been amended. Claims 22-27 are 
new. Claims 8-27 are pending and have been considered below. 

To expedite prosecution, the applicant should consider MPEP paragraph 714.12: 

"Many of the difficulties encountered in the prosecution of patent applications after final 
rejection may be alleviated if each applicant includes, at the time of filing or no later than 
the first reply, claims varying from the broadest to which he or she believes he or she is 
entitled to the most detailed that he or she is willing to accept." 

Response to Arguments 

1 . Applicant's arguments filed 13 April 2009 have been fully considered but they are 
not persuasive. 

The applicant argues "that modern windowing programs normally cascade, meaning 
that when a new window is opened, the new window will be shifted down and to the 
right from the previous window. So, unless exactly the right number of windows have 
been opened prior to the start of the Tuttle testing system, Tuttle will reject a perfectly 
fine operation due to the shifting resulting from cascading windows. 
In addition, since Tuttle is only able to match a screen for "pass/fail regression testing" 
(Tuttle Col 14, lines 38-41), there is no opportunity to choose what action to take, 
depending upon what occurs. Instead, Tuttle must simply halt the program and return an 
error when a screen image fails to match. 

The Examiner will appreciate that Tuttle is consequently limited solely to testing of 
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hardware such as might occur at a manufacturing facility, where every computer is 
loaded with the same software, and is tested immediately at start-up. 
In contrast, the present invention searches the image for individual elements, such as 
"icons as illustrated, or various buttons, message boxes, prompts or any other graphical 
image or control." (Present specification, page 8, lines 17-19) As a result, the user does 
not need to worry where on the screen the image appears, and can make intelligent 
decisions about any subsequent actions, including the placement of a mouse click or 
other similar action relative to the placement of the individual element, and also 
alternatives that might arise and still be handled to continue the testing. 
This may seem to be a subtle difference, but the implications are far reaching. For 
exemplary purposes, a calculator can be tested through all numbers and operations by 
storing the images for each of the operations and the numbers 0 - 9. For a simple 
calculator, the operations might be "add", "subtract", "multiply", "divide", "equals", and 
"clear". Then, using simple "For Next", "Repeat With", "Do While" loops or the like, each 
of the operations may be tested and the results confirmed by saving the images of 
operations, numerals and any other necessary elements, and using very simple loops 
requiring less than forty lines of code. Both the code and the images will take up very 
little storage space. This is done without concern for where on the screen the calculator 
might appear. In contrast, Tuttle must actually execute every possible test, and then 
save the image to compare with future systems. The storage of these images will 
require an impractical amount of time, and an unreasonable amount of storage space." 
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The applicant seems to be making a conceptual argument and has not cited any claim 
language for which the applicant believes that Tuttle is deficient. In response to 
applicant's argument that the references fail to show certain features of applicant's 
invention, it is noted that the features upon which applicant relies are not recited in the 
rejected claim(s). Although the claims are interpreted in light of the specification, 
limitations from the specification are not read into the claims. See In re Van Geuns, 988 
F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). Furthermore, the applicant is reminded 
that, as per 35 USC Paragraph 112, "The specification shall conclude with one or more 
claims particularly pointing out and distinctly claiming the subject matter which the 
applicant regards as his invention." Therefore, the applicant must present arguments 
traversing the examiner's rejection of the claims, not the conceptual differences 
between the examiner's cited prior art and the applicants invention as described in the 
written description. Therefore, the examiner believes that the prior art of record as cited 
discloses the limitations as claimed. 



Claim Rejections - 35 USC § 103 

2. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

3. Claims 8-21 are rejected under 35 U.S.C. 103(a) as being unpatentable over 



Tuttle et al. (US 5,157,782) in view of Moore et al. (US 5,604,509). 
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Claim 1: (Canceled) 
Claim 2: (Canceled) 
Claim 3: (Canceled) 
Claim 4: (Canceled) 
Claim 5: (Canceled) 
Claim 6: (Canceled) 
Claim 7: (Canceled) 



Claims 8 and 16: Tuttle discloses method for using a first computer system to remotely 
monitor and interact with the operation of a second computer system through a 
graphical user interface of said second computer system, comprising the steps of: 

a. receiving a pixel image said second computer system graphical user interface at 
said first computer system (column 6, lines 11-21, column 8, lines 39-47); 

b. searching said pixel image of said second computer system graphical user 
interface for a first graphical element contained within and comprising less than 
said pixel image (column 8, lines 39-47), 

c. generating a user peripheral input device input action within said second 
computer system graphical user interface as interpreted by a second computer 
peripheral input device controller channel by passing a signal from said first 
computer system to said second computer system graphical user interface 
(column 6, lines 1-10, 22-28); 
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d. monitoring said second computer system graphical user interface or a pixel 
image of said second computer system graphical user interface from said first 
computer system for an expected second graphical element (column 6, lines 28- 
30, 34-39); and 

e. signaling a failure at said first computer system if said expected second graphical 
element is not detected (column 6, lines 39-41). 

but does not disclose 

a. generating a user peripheral input device input action within said second 
computer system graphical user interface as interpreted by a second computer 
peripheral input device controller channel by passing a signal through an i/o 
communications channel from said first computer system to said second 
computer system graphical user interface responsive to said receiving and 
searching steps, 

b. monitoring said pixel image of said second computer system graphical user 
interface from said first computer system for an expected second graphical 
element contained within and comprising less than said pixel image within a 
predetermined time interval; and 

c. signaling a failure at said first computer system if said predetermined time 
interval elapses without detecting said expected second graphical element. 
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However, Moore discloses a system where video data is sent from a second computer 
(a system under test) to a first computer (host or remote computer) over a 
communication channel (column 2, lines 41-59) and displayed on a display of the first 
computer (column 3, lines 8-19). A user inputs, i.e. mouse and keyboard inputs, can be 
sent over a communications channel from the first (host or remote computer) to the 
second computer, system under test (column 4, lines 16-23) where the input data is 
processed (column 4, lines 39-50). Therefore, it would have been obvious to one 
having ordinary skill in the art at the time of the invention to add generating a user 
peripheral input device input action within said second computer system (system under 
test) graphical user interface as interpreted by a second computer peripheral input 
device controller channel by passing a signal through an i/o communications 
channel from said first computer system (host or remote computer) to said second 
computer system graphical user interface responsive to said receiving step to Tuttle. 
One could have been motivated to add generating a user peripheral input device input 
action within said second computer system graphical user interface as interpreted by a 
second computer peripheral input device controller channel by passing a signal 
through an i/o communications channel from said first computer system to said 
second computer system graphical user interface responsive to said receiving step to 
Tuttle in order to extend the range of communication between the first and second 
computers of Tuttle. It appears that Tuttle is intended to be used wherein the first and 
second computers are in close proximity. Adding the limitations of Moore to Tuttle could 
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allow Tuttle to be able to test the software/hardware of the second computer at the first 
computer over a network over a larger distance. 

Neither Tuttle nor Moore disclose generating a user peripheral input device input action 
within said second computer system graphical user interface as interpreted by a second 
computer peripheral input device controller channel by passing a signal through an i/o 
communications channel from said first computer system to said second computer 
system graphical user interface responsive to said receiving step. However, it would 
have been obvious to one having ordinary skill in the art at the time of the invention to 
add generating a user peripheral input device input action within said second computer 
system graphical user interface as interpreted by a second computer peripheral input 
device controller channel by passing a signal through an i/o communications channel 
from said first computer system to said second computer system graphical user 
interface responsive to said receiving step to Tuttle and Moore. One could have 
been motivated to add generating a user peripheral input device input action within said 
second computer system graphical user interface as interpreted by a second computer 
peripheral input device controller channel by passing a signal through an i/o 
communications channel from said first computer system to said second computer 
system graphical user interface responsive to said receiving step to Tuttle and Moore 
because as a user in Tuttle issues inputs that are communicated to the second 
computer and the second computer responds to the inputs with video signals, it is 
reasonable to conclude that a user's subsequent input(s) would be in response to video 
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responses by the second computer as is typical when a user uses a user interface (Ul) 
of a computer. That is, when a user engages a Ul of a computer, the user's inputs are 
neither random nor disassociated from the response to the input. Therefore, it is 
reasonable to conclude that in testing a Ul or a computer, a user's input would be 
associated in some way, either in response to or in anticipation of, a Ul response. 

Neither Tuttle nor More disclose: 

a. monitoring said second computer system graphical user interface or a pixel 
image of said second computer system graphical user interface from said first 
computer system for an expected second graphical element within a 
predetermined time interval; and 

b. signaling a failure at said first computer system if said predetermined time 
interval elapses without detecting said expected second graphical element. 

However, Tuttle discloses that at those points during playback which corresponds to the 
original command to capture visual display data, the software modules send the digital 
video signal processing unit (DVPU) the previously stored visual display signals, and 
the DVPU then captures new visual display signals from the visual display device. The 
previously stored visual display signals are compared by the DVPU to the newly 
captured visual display signals and if these signals do not match, then the DVPU sends 
an indication to the software modules on the host that an error has occurred (column 6, 
lines 31-41) and another such parameter used primarily during Playback Mode sets the 
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amount which the software modules waits for a response from the DVPU/SUT; before 
the comparison test is deemed to have failed (column 39, lines 53-57). Therefore, it 
would have been obvious to one having ordinary skill in the art at the time of the 
invention to add 

a. monitoring said second computer system graphical user interface or a pixel 
image of said second computer system graphical user interface from said first 
computer system for an expected second graphical element within a 
predetermined time interval; and 

b. signaling a failure at said first computer system if said predetermined time 
interval elapses without detecting said expected second graphical element, 

to Tuttle and Moore. One could have been motivated to add 

a. monitoring said second computer system graphical user interface or a pixel 
image of said second computer system graphical user interface from said first 
computer system for an expected second graphical element within a 
predetermined time interval; and 

b. signaling a failure at said first computer system if said predetermined time 
interval elapses without detecting said expected second graphical element, 

to Tuttle and Moore because It is unreasonable to conclude that a user in Tuttle would 
randomly intersperse capture commands through an input sequence. In fact, the 
examiner asserts that a user would initiate a capture command where it would be most 
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appropriate to capture the video results of an input or a sequence of inputs. In other 
words, a user would only issue a capture when the user knew that an input or sequence 
of inputs would result in an expected video signal, i.e. the user would not issue the 
capture command before the expected resultant video signal. However, it is also 
unreasonable to expect that the user would wait an indeterminate amount of time after 
the input or sequence of inputs to issue a capture command. Therefore, it would have 
been obvious to include a timing mechanism in the capture command such that the 
capture command is issued before after the expected resultant video signal is 
generated but within a reasonable amount of time. 

Claims 9 and 17: Tuttle and Moore disclose the methods of claims 8 and 16 and Tuttle 
further discloses the steps of: 

a. transferring said user input action to a script stored on said first computer system 
(column 7, lines 19-30); 

b. re-executing said steps of receiving, generating, monitoring and signaling 
subsequent to said storing step under control of said stored script (column 7, 
lines 48-58).. 

Claims 1 0 and 18: Tuttle and Moore disclose the methods of claims 8 and 1 6 and Tuttle 
discloses further the steps of: 

a. providing graphical user interface language extensions commands to a scripting 
language (column 15, lines 56-68, column 16, lines 1-24); and 
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b. passing said generated user input action through said graphical user interface 
language extensions from a scripting language processor to a language 
extensions processor (column 8, lines 21-26). 

Claims 1 1 and 1 9: Tuttle and Moore disclose the methods of claims 8 and 1 6 and Tuttle 
further discloses the steps of: 

a. generating a user input action within said second computer system responsive to 
said second graphical element (column 8, lines 18-26); 

b. monitoring said second computer system graphical user interface for an 
expected third graphical element contained within and comprising less than said 
pixel image within a predetermined time interval (column 6, lines 28-30, 34-39, 
column 8, lines 39-47); and 

c. signaling a failure at said first computer system if said predetermined time 
interval elapses without detecting said expected third graphical element (column 
6, lines 39-41, column 39, lines 53-57). 

Claims 12 and 20: Tuttle and Moore disclose the methods of claims 8 and 16 and Tuttle 
further discloses the steps of: 

a. receiving a local user input action at said first computer system within said local 
display (column 6, lines 1-4, column 36, lines 6-14); 

b. wherein said generated user input action emulates said local user input action 
(column 7, lines 12-18), 
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but does not disclose depicting said second computer system graphical user interface 
upon a local display of said first computer system including said first graphical element. 
However, Moore discloses depicting said second computer system graphical user 
interface upon a local display of said first computer system including said first graphical 
element (column 3, lines 8-25). Further, Turtle discloses that the first computer (host) 
has direct access to the digital video data of the display screen of the second computer 
(system under test) (column 13, lines 57-59), and inputting inputs through a touch 
screen (column 36, lines 6-14). Therefore, it would have been obvious to one having 
ordinary skill in the art at the time of the invention to add depicting said second 
computer system graphical user interface upon a local display of said first computer 
system including said first graphical element to Turtle and Moore. One could have been 
motivated to add depicting said second computer system graphical user interface upon 
a local display of said first computer system including said first graphical element to 
Turtle and Moore because as a user in Turtle issues inputs that are communicated to 
the second computer and the second computer responds to the inputs with video 
signals, it is reasonable to conclude that a user's subsequent input(s) would be in 
response to video responses by the second computer as is typical when a user uses a 
user interface (Ul) of a computer. That is, when a user engages a Ul of a computer, the 
user's inputs are neither random nor disassociated from the response to the input. 
Therefore, it is reasonable to conclude that in testing a Ul or a computer, a user's input 
would be associated in some way, either in response to or in anticipation of, a Ul 
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response. In order for a user to issue commands in response to Ul responses on the 
second computer, it would be beneficial for the user to be able to see the display of the 
second computer on the display of the first computer. 

Claims 13 and 21 : The method of claim 8 further comprising the steps of: 

a. providing graphical user interface language extensions commands to a scripting 
language (column 15, lines 56-68, column 16, lines 1-24); and 

b. receiving a local user input action within said local display (column 6, lines 1-4, 
column 36, lines 6-14); 

c. transferring said user input action to a script stored on said first computer system 
(column 7, lines 19-30); 

d. passing said generated user input action through said graphical user interface 
language extensions from a scripting language processor to a language 
extensions processor for reproduction at said second computer system graphical 
user interface, wherein said generated user input action emulates said local user 
input action (column 15, lines 56-68, column 16, lines 1-24, column 8, lines 21- 
26, column 7, lines 12-18); and 

e. re-executing said steps of receiving, searching, generating, monitoring and 
signaling subsequent to said storing step under control of said stored script 
(column 7, lines 48-58, column 8, lines 39-47), 
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and Moore discloses depicting said computer system graphical user interface upon a 
local display of said first computer system including said first graphical element (column 
3, lines 8-25). Further, Tuttle discloses that the first computer (host) has direct access 
to the digital video data of the display screen of the second computer (system under 
test) (column 13, lines 57-59), and inputting inputs through a touch screen (column 36, 
lines 6-14). Therefore, it would have been obvious to one having ordinary skill in the art 
at the time of the invention to add depicting said computer system graphical user 
interface upon a local display of said first computer system including said first graphical 
element to Tuttle and Moore. One could have been motivated to add depicting said 
computer system graphical user interface upon a local display of said first computer 
system including said first graphical element to Tuttle and Moore because as a user in 
Tuttle issues inputs that are communicated to the second computer and the second 
computer responds to the inputs with video signals, it is reasonable to conclude that a 
user's subsequent input(s) would be in response to video responses by the second 
computer as is typical when a user uses a user interface (Ul) of a computer. That is, 
when a user engages a Ul of a computer, the user's inputs are neither random nor 
disassociated from the response to the input. Therefore, it is reasonable to conclude 
that in testing a Ul or a computer, a user's input would be associated in some way, 
either in response to or in anticipation of, a Ul response. In order for a user to issue 
commands in response to Ul responses on the second computer, it would be beneficial 
for the user to be able to see the display of the second computer on the display of the 
first computer. 
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Claim 14: Tuttle discloses a method for enabling a local system to remotely operate a 
remote computer system through a graphical user interface on said remote computer 
system by using local scripts that selectively respond to changes in graphical displays 
upon said graphical user interface of said remote computer system, comprising the 
steps of: 

a. capturing user input effected in said depiction of said remote system graphical 
user interface display (column 6, lines 22-28); 

b. implementing through a local system command language set user input 
emulations representative of said captured user input reproduced at said remote 
computer system graphical user interface through a channel (column 15, lines 
56-68, column 16, lines 1-24); 

c. image processing said remote computer system graphical displays to detect a 
first entity contained within and comprising less than said graphical display upon 
said graphical user interface of said remote computer system (column 6, lines 
28-30, 34-39); 

d. controlling a flow of execution of said local system through a scripting language 
having scripting commands in combination with said command language set 
(column 6, lines 31-48, column 8, lines 39-47); and 

e. communicating between said local system and said remote computer system 
graphical user interface (column 6, lines 31-48). 
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Moore discloses displaying a depiction of said remote system graphical user interface 
display on said local system (column 3, lines 8-25). Further, Tuttle discloses that the 
first computer (host) has direct access to the digital video data of the display screen of 
the second computer (system under test) (column 13, lines 57-59), and inputting inputs 
through a touch screen (column 36, lines 6-14). Therefore, it would have been obvious 
to one having ordinary skill in the art at the time of the invention to add displaying a 
depiction of said remote system graphical user interface display on said local system to 
Tuttle. One could have been motivated to add displaying a depiction of said remote 
system graphical user interface display on said local system to Tuttle because as a user 
in Tuttle issues inputs that are communicated to the second computer and the second 
computer responds to the inputs with video signals, it is reasonable to conclude that a 
user's subsequent input(s) would be in response to video responses by the second 
computer as is typical when a user uses a user interface (Ul) of a computer. That is, 
when a user engages a Ul of a computer, the user's inputs are neither random nor 
disassociated from the response to the input. Therefore, it is reasonable to conclude 
that in testing a Ul or a computer, a user's input would be associated in some way, 
either in response to or in anticipation of, a Ul response. In order for a user to issue 
commands in response to Ul responses on the second computer, it would be beneficial 
for the user to be able to see the display of the second computer on the display of the 
first computer. 
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Moore discloses a system where video data is sent from a second computer (a system 
under test) to a first computer (host or remote computer) over a communication channel 
(column 2, lines 41-59) and displayed on a display of the first computer (column 3, lines 
8-19). A user inputs, i.e. mouse and keyboard inputs, can be sent over a 
communications channel from the first (host or remote computer) to the second 
computer, system under test (column 4, lines 16-23) where the input data is processed 
(column 4, lines 39-50). Therefore, it would have been obvious to one having ordinary 
skill in the art at the time of the invention to add communicating between said local 
system and said remote computer system graphical user interface through a 
communication interface responsive to said flow controlling step to Tuttle and Moore. 
One could have been motivated to add communicating between said local system and 
said remote computer system graphical user interface through a communication 
interface responsive to said flow controlling step to Tuttle and Moore in order to extend 
the range of communication between the first and second computers of Tuttle. It 
appears that Tuttle is intended to be used wherein the first and second computers are in 
close proximity. Adding the limitations of Moore to Tuttle could allow Tuttle to be able to 
test the software/hardware of the second computer at the first computer over a network 
over a larger distance. 

Neither Tuttle nor Moore disclose controlling a flow of execution of said local system 
through a scripting language having scripting commands in combination with said 
command language set responsive to a detection of said first entity during said 
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image processing step. However, it would have been obvious to one having ordinary 
skill in the art at the time of the invention to add controlling a flow of execution of said 
local system through a scripting language having scripting commands in combination 
with said command language set responsive to a detection of said first entity during 
said image processing step to Tuttle and Moore. One could have been motivated to 
add controlling a flow of execution of said local system through a scripting language 
having scripting commands in combination with said command language set 
responsive to a detection of said first entity during said image processing step to 
Tuttle and Moore because as a user in Tuttle issues inputs that are communicated to 
the second computer and the second computer responds to the inputs with video 
signals, it is reasonable to conclude that a user's subsequent input(s) would be in 
response to video responses by the second computer as is typical when a user uses a 
user interface (Ul) of a computer. That is, when a user engages a Ul of a computer, the 
user's inputs are neither random nor disassociated from the response to the input. 
Therefore, it is reasonable to conclude that in testing a Ul or a computer, a user's input 
would be associated in some way, either in response to or in anticipation of, a Ul 
response. 

Claim 15: Tuttle and Moore disclose the method for enabling a local system to remotely 
operate a remote computer system through a graphical user interface on said remote 
computer system of claim 14 and Tuttle further discloses the steps of: 

a. storing said scripting commands into a storing means (column 7, lines 23-26); 
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b. inserting a command from said command language set into said storing means 
(column 15, lines 56-63, column 29, lines 23-28); and 

c. executing said inserted stored command (column 29, lines 29-39). 



Claim 22: Tuttle and Moore disclose the method of claim 8, and Tuttle further discloses 
said step of generating a user peripheral input device input action further comprises 
locating said user peripheral input device input action within said first graphical element, 
(column 39, lines 9-20). 

Claim 23: (New Tuttle and Moore disclose the method of claim 22, and Tuttle further 
discloses said user peripheral input device input action further comprises a click event 
(column 39, lines 9-20). 

Claim 24: Tuttle and Moore disclose the method of claim 14, and Tuttle further discloses 
said step of communicating between said local system and said remote computer 
system graphical user interface further comprises locating a user peripheral input device 
input action at a location relative to said first entity column (column 39, lines 9-20). 



Claim 25: Tuttle and Moore disclose the method of claim 24, and Tuttle further discloses 
said user peripheral input device input action further comprises a click event (column 
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39, lines 9-20). 

Claim 26: Tuttle and Moore disclose the method of claim 16, and Tuttle further discloses 
said step of generating a user peripheral input device input action further comprises 
locating said user peripheral input device input action within said first graphical entity 
and Tuttle further discloses (column 39, lines 9-20). 

Claim 27: Tuttle and Moore disclose the method of claim 26, and Tuttle further discloses 
said user peripheral input device input action further comprises a click event (column 
39, lines 9-20). 



Conclusion 

4. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 .136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
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the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to JOHN M. HEFFINGTON whose telephone number is 
(571)270-1696. The examiner can normally be reached on Mon - Fri 8:00 - 5:30 EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Weilun Lo can be reached on (571) 272-4847. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

JMH 



/Steven B Theriault/ 

Primary Examiner, Art Unit 2179 



